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ABSTRACT 

In this paper, we developed two kinds of application software run on a mobile/wearable device for autistic spectrum 
disorder students, intellectual disability students, or physically challenged. One of the applications is expression 
detector/evaluator using a smartphone and a small expression sensor for social skill training. This sensor can inspect 
human’s face-expression. We developed simple games with using it. Students can train to make his/her own expression 
by playing the game. The second of them is mental detector application using a heartbeat sensor in a smart watch. Using 
it, a students can predict own agitation by his/herself and the application informs it to a teacher. These two approach are 
desirable ICT application examples for the person. These can lead the students self-respecting, teachers mentioned. 
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1. INTRODUCTION AND OUR WORKS 

Japan government enforced the new law of reasonable consideration to any disorder/disability students in any 
education from April 2016. However, teachers who don’t have enough experience in special needs education 
aren’t able to grasp and infer such students’ intention and feeling unfortunately. Moreover, students have the 
desire to use their bodies effectively. In particular, face expression is one of the important factors for 
communication with others. To respond to these issues, we developed two approaches. 

The first approach is a face-expression training game for students using the expression-sensing sensor 
OMRON HVC-C (Human Vision Components C1B model) and a smartphone (Figure 1). This game is a kind 
of quiz game. A student answers questions by making face-expression e.g. smile, angry, sad, surprise and 
serious. This application is able to record data which corresponds to the duration of time and rate of correct 
answers. After training, the screen displays face-expressions in the same sequence. Appling this approach, a 
person who is not only physically challenged and but also has a panic disorder will be able to send a signal or 
control devices. 



Figure 1. A face sensing sensor HVC-C (left) and a question mode screen on a smart -phone (right) 
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The second approach is a self-recognition application using a smart-watch and a smartphone. The 
smart- watch is a watch-type wearable device that resembles a wristwatch. In this work, we use a smart- watch 
“Moto 360”. Using it, we can measure the rate of heartbeats easily. According to the rate, this application can 
show meaningful messages on the screen of the smart-watch e.g. “You seem to be sad. Are you OK?”, “Calm 
down” or “Breathe deeply” and so on. At the same time, the result is sent to a teachers’ tablet PC who can 
then grasp the situation and take appropriate action. 

Figure 2 shows a user wearing the smart- watch on the wrist and running the application. The application 
can indicate notices such as vibration, showing messages/images on the device’s screen when the rate of 
heartbeat exceeds a particular value which was set in advance. When rate of the heartbeat falls, the reason is 
because it is thought that I am in a tense state during an angry time. 



Figure 2. Wearing a smart-watch on the wrist 


2. HOW TO USE OUR APPLICATION 

Firstly, our developed face-expression training application uses a smartphone and the HVC-C. After the 
application is installed on the smartphone, connect the smartphone and the HVC-C. When both are connected, 
the screen of the smartphone turns into a question mode, that shows a situation and ask what type of 
expression a student makes at that time. If the student decided to make a face-expression, the application 
evaluates the face-expression by touching the smartphone screen. If the student can select the correct 
face-expression, the smartphone screen transitions to the next question mode (Figure3). During making 
face-expression, the smartphone records some data at that moment until the correct face-expression and name 
of face-expressions is selected. After training, a smartphone creates a data file that contains the duration of 
time, a name of face-expression and so on. To share the data files, people who want to know a result of 
training can see it with other devices e.g. smartphone, tablet and computer. 



Figure 3. Transit screen on a smartphone 

Secondly, we developed a self-recognition application. A person who wants to measure heartbeat rate 
wears the smart- watch on the wrist. To start the application on a smartphone, you need touch a button on the 
smart- watch application. It begins the measurement of the heartbeat. Measuring the rate of heartbeat with the 
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smart-watch application, the smart-watch sends this data to the smartphone. On the smartphone side, the 
smartphone draws a graph on screen using the received heartbeat rate data. The graph is capable of 
enlargement, reduction and movement, because when you want to watch whole or detail of the graph, 
expansion reduction and movement to the side are necessary. Figure 4 displays the numerical value that I 
measured with a smart watch as a graph to the screen of the smartphone. We think that the raise and downs of 
heartbeat may be normal by making the value that I measured a graph. In addition, smartphone creates a data 
file of the heartbeat rate. This data file can be accessed by other applications and devices such as another 
smartphone, computer and so on. It is able to share heartbeat data and see the same graph or data. 



Figure 4. A graph of captured heartbeat on a smartphone screen 


3. DISCUSSION 

We think that these two kinds of applications that we developed may be useful for students with any 
disorder/disability. For example, in the case of the student where it is difficult for them to understand feelings 
by the expressions of other people, and for metacognition of one’s own expression, the student can practice 
making some expressions and the teacher can see the students expression. In addition, in the case of the 
student that it is difficult to take communication to other people and to grasp feeling of other people, teacher 
remember the situation that the student has a heightened feeling and calm the student down. 


4. CONCLUSION 

We developed two kinds of application software to run on a mobile/wearable device for the student who 
needed special support. The first application allows the user to practice making face-expressions in accord 
with particular situations. We think that they can practice making face-expressions and read feelings from 
face-expression through the connection between face-expression and feelings by using this application. The 
second application is to measure the rate of heartbeat. We think a teacher can understand the situation of 
heightened feeling of the student who is difficult to communicate with by using this application. In addition, 
we think that a teacher can take prior measure because a teacher will be able to grasp the situation when a 
student is exited under control. 

We think that these two applications help students who have any disorder/disability in which it is difficult 
to read the feeling of other people, to make face-expression and to take communication. 
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